Strong Markov Property

\(n\)-step transition

Definition 1. For a time homogeneous Markov chain \(X:\Omega\to\sX^{\Z_+}\), we can define \(n\)-step transition probability matrix \(P^{(n)}\), with its \((x,y)\) entry being the \(n\)-step transition probability for \(X_{m+n}\) to be in state \(y\) given the event \(\set{X_m = x}\). That is, \(p_{xy}^{(n)} \triangleq P(\set{X_{n+m} = y }|\set{ X_m = x})\) for all \(x,y \in \sX\) and \(m,n \in \Z_+\).

Remark 1. That is, the row \(P^{(n)}_x = (p^{(n)}_{xy}: y \in \sX) \in \cM(\sX)\) is the conditional distribution of \(X_n\) given the initial state \(\set{X_0 = x}\).

Theorem 2. The \(n\)-step transition probabilities for a homogeneous Markov chain form a semi-group. That is, for all positive integers \(m,n \in \Z_+\)

Proof. Proof. The events \(\set{ \set{X_m = z}: z \in \sX}\) partition the sample space \(\Omega\), and hence we can express the event \(\set{X_{m+n} = y}\) as the following disjoint union It follows from the Markov property and law of total probability that for any states \(x, y\) and positive integers \(m, n\) Since the choice of states \(x,y \in \sX\) were arbitrary, the result follows. ◻

Corollary 3. The \(n\)-step transition probability matrix is given by \(P^{(n)} = P^n\) for any positive integer \(n\).

Proof. Proof. In particular, we have \(P^{(n+1)} = P^{(n)}P^{(1)} = P^{(1)}P^{(n)}\). Since \(P^{(1)} = P\), we have \(P^{(n)} = P^n\) by induction. ◻

Definition 4. For a time homogeneous Markov chain \(X:\Omega\to\sX^{\Z_+}\) we denote the probability mass function of Markov chain at step \(n\) by \(\pi_n\in \cM(\sX)\).

Lemma 5 (Chapman Kolmogorov). The right multiplication of a probability vector with the transition matrix \(P\) transforms the probability distribution of current state to probability distribution of the next state. That is,

Proof. Proof. To see this, we fix \(y \in \sX\) and from the law of total probability and the definition conditional probability, we observe that ◻

Strong Markov property (SMP)

We are interested in generalizing the Markov property to any random times. For a DTMC \(X: \Omega \to \sX^{\Z_+}\) and a random variable \(\tau: \Omega \to \N\), we are interested in knowing whether for any historical event \(H_{\tau-1} = \cap_{n = 0}^{\tau-1}\set{X_n = x_n}\) and any state \(x, y \in \sX\), we have

Example 6 (Two-state DTMC). Consider the two state Markov chain \(X \in \set{0, 1}^{\Z_+}\) such that \(P_0\set{X_1 = 1} = q\) and \(P_1\set{X_1=0} = p\) for \(p,q \in [0,1]\). Let \(\tau : \Omega \to \N\) be a random variable defined as That is, \(\set{\tau = n} = \set{X_1 = 0, \dots, X_n = 0, X_{n+1} = 1}\). Hence, for the historical event \(H_{\tau-1} = \set{X_1 = \dots, X_{\tau-1} = 0}\), the conditional probability \(P(\set{X_{\tau+1}= 1}\mid H_{\tau-1}\cap\set{X_\tau = 0}) = 1\), and not equal to \(q\).

Definition 7. Let \(\tau:\Omega\to\N\) be a stopping time with respect to a random sequence \(X:\Omega\to\sX^{\Z_+}\). Then for all states \(x,y \in \sX\) and the event \(H_{\tau-1} = \cap_{n=0}^{\tau-1}\set{X_n = x_n}\), the process \(X\) satisfies the strong Markov property if

Lemma 8. Homogeneous Markov chains satisfy the strong Markov property.

Proof. Proof. Let \(X:\Omega \to \sX^{\Z_+}\) be a homogeneous DTMC with transition matrix \(P\), and \(\tau:\Omega\to\N\) be an associated stopping time. We take any historical event \(H_{\tau-1} = \cap_{n = 0}^{T-1}\set{X_n = x_n}\), and states \(x, y \in \sX\). From the definition of conditional probability, the law of total probability, and the Markovity of the process \(X\), we have This equality follows from the fact that the event \(\set{\tau= n}\) is completely determined by \((X_0, \dots, X_n)\). ◻

Remark 2. Consider a homogeneous DTMC \(X:\Omega\to\sX^{\Z_+}\) and the first instant \(\tau_k \triangleq \tau_X^{\set{y},k}\) for the process \(X\) to hit \(k\) times, a state \(y\in\sX\). Recall that \(\tau_0 \triangleq 0\) and recurrence time \(H_k \triangleq \tau_k - \tau_{k-1} = \inf\set{n \in \N: X_{\tau_{k-1}+n}= y}\) for all \(k \in \N\). We define a process \(Y:\Omega\to\sX^{\Z_+}\) where \(Y_m \triangleq X_{\tau_k + m}\) for all \(m \in \Z_+\). If \(\tau_k\) is almost surely finite, then it is a stopping time with respect to process \(X\). Using strong Markov property of DTMC \(X\), we will show that \(Y\) is a stochastic replica of \(X\) with \(X_0 = y\).

Hitting and Recurrence Times

We will consider a time-homogeneous discrete time Markov chain \(X: \Omega \to \sX^{\Z_+}\) on countable state space \(\sX\) with transition probability matrix \(P: \sX \times \sX \to [0,1]\), and initial state \(X_0 = x \in \sX\). We denote the natural filtration generated by the process \(X\) as \(\sF_\bullet\), where \(\sF_n \triangleq \sigma(X_0, \dots, X_n)\) for all \(n \in \N\).

Remark 3. Starting from state \(x\), the mean number of visits to state \(y\) in \(n\) steps is \(\E_xN_y(n) = \sum_{k=1}^np_{xy}^{(k)}\). From the monotone convergence theorem, we also get that \(E_xN_y(\infty) = \sum_{k \in \N}p_{xy}^{(k)}\).

Remark 4. If \(\tau_{k-1}\) is almost sure finite, then \(\tau_{k-1}\) is a stopping time for process \(X\). From the strong Markov property of homogeneous DTMC \(X\) applied to stopping time \(\tau_{k-1}\), it follows that the future \(\sigma(X_{\tau_{k-1}+j}: j \in \N)\) is independent of the past \(\sigma(X_0, \dots, X_{\tau_{k-1}})\) given the present \(\sigma(X_{\tau_{k-1}})\). Since \(X_{\tau_{k-1}}= y\) for \(k \ge 2\) deterministically, it follows that \(\sigma(X_{\tau_{k-1}})\) is a trivial event space and the future \(\sigma(X_{\tau_{k-1}+j}: j \in \N)\) is independent of the random past \(\sigma(X_0, \dots, X_{\tau_{k-1}})\). We further observe that the distribution of \(\sigma(X_{\tau_{k-1}+j}: j \in \N)\) is identical to distribution of \(X\) given \(X_0= y\). Thus, the process \((X_{\tau_{k-1}+j}: j \in \N)\) is distributed identically for all \(k \ge 2\).

Remark 5. We observe that the recurrence time satisfies \(\set{H_k = n} \in \sigma(X_{\tau_{k-1}+j}: j \in [n])\) for all \(n \in \N\), and hence the recurrence time \(H_k\) is independent of the random past \(\sigma(X_0, \dots, X_{\tau_{k-1}})\). Recursively applying this fact, we can conclude that \((H_1, \dots, H_k)\) are independent random variables. Further, since \((X_{\tau_{k-1}+j}: j \in \N)\) is distributed identically for all \(k \ge 2\), it follows that \((H_k: k \ge 2)\) are distributed identically.

Lemma 9. If \(H_1\) and \(H_2\) are almost surely finite, then the random sequence \((H_k: k \ge 2)\) is .

Proof. Proof. From the above two remarks, it suffices to show that each term of the random sequence \(\tau:\Omega\to\N^\N\) is almost surely finite. We will show this by induction. Since \(\tau_1 = H_1\) is almost surely finite, it follows that \(\tau_1\) is stopping time. Since \(\tau_2 = \tau_1+ H_2\) is almost surely finite, it follows that \(\tau_2\) is a stopping time. By inductive hypothesis \(\tau_{k-1}\) is almost surely finite, and hence \(H_k\) is independent of \((H_1, \dots, H_k)\) and identically distributed to \(H_2\) and is almost surely finite. It follows that \(\tau_k = \tau_{k-1} + H_k\) is almost surely finite, and the result follows. ◻